DOCOHENT RESQHE 



BD 091 992 



HE 005 551 



XOTHOR 
TITLE 

PUB DATE 
NOTE 



EDRS PRICE 
DESCRIPTORS 



Preeaani Donald J.; Niemeyer, Roger C. 
The iBpact .of Written Comments on student 
Achievenent • 
74 

lOp.; Paper presented at the Annual Meeting of the 
American Educational Research Association (59th, 
Chicagbr Illinois, April 1974) 

BF-$0,75 HC-$1*50 PLUS POSTAGE 

♦Academic Achievement; College Faculty; ♦Graduate 
Students; Graduate Study; ♦Higher Education; 
Perfornance Criteria; Performance Pactors; Research 
Projects; ♦Teacher Influence; ♦Teacher Response; 
Tests 



ABSTRACT 

This study sought to determine if an instructor of a 
cottpetency-based course who sets performance standards at a 
comparatively low level might facilitate higher levels of student 
achievement through the use of written comments on unit posttests. 
Eighty-eight >^tudents in a graduate education course were randomly 
assigned to one of four experimental groups* Group 1 received 
comments on five posttests that encouraged them to adopt a higher 
"subjective" performance standard than that required in the course* 
Group 2 received similar comments on three posttests* Group 3 
received standard comments on five posttests. Group ^ received no 
comments* Differences among the four groups on three measures of 
achievement were not statistically significant * (Author) 
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THE IMPACT OF WRITTEN COMMENTS ON STUDENT ACHIEVEMENT 

by 

Dr. Donald J. Freeman, Michigan State University 
Dr. Roger C. Niemeyer, Michigan State University 

- ABSTRACT • 



This study sought to determine if an instructor of a cotnpetency-based 
course who sets performance standards at a comparatively low level might 
facilitate higher levels of student achievement through the use of written 
comments on unit postests. 88 Ss in a graduate education course were 
randomly assigned to one of four experimental groups. Group I received 
cooments on five postests which encouraged them to adopt a higher /^subjective" 
performance standard than that required in the course. Group II received 
similar comments on three postests. Group III received standard comments on 
five postests* Group IV recelved.no comments* Differences among the four 
groups on three measures of achievement were not statistically significant. 



INTRODUCTION 



\ 



Whenever an instructor uses a mastery approach, he must decide what 
criterion level he will adopt for each unit of instruction. Although techniques 
for computing an "optimal standard" have been proposed, most would agree that 
this decision to a large extent represents an arbitrary Judgement (Cureton, 1971; 
Ebel, 1971] Millman, 1970 and 1973) * Following a review of five general compu- 
tational techniques , for example, Millman concludes . • they all required the 
use of judgement at some stage of their execution." (Millman, 1973, p,214). 

However, careful consideration of various relevant factors (example: Are 
the skills necessary for future learning?) may make decisions regarding an appro- 
priate standard somewhat more meaningful. The problem confronting an instructor 
of an in-service education course is that two of these factors appear to conflict. 
On the one hand, an Instructor wants to set the criterion level for each unit at 
a realistic level whereby most students (active teachers) will be successful in 



passing unit postests on their first or second attempt . A teacher who 
repeatedly falls several unit postests is not likely to attempt to Implement 
the Ideas presented in those units In his/her teaching. On the other hand, 
Informal feedback from former students and some research evidence suggests that 
the overall achievement of students may suffer when unit standards are set at 
a comparatively low level (Blumfeld, et ,al. , 1971) . 

One way in which this apparent conflict might be resolved Is to set unit 
standards at a realistic levd and press for higher levels of student achieve- 
ment through other Instructional means. One convenient option might be to 
provide written comments on unit postests. This instructional technique has 
been shown to Influence test performance , in other Instructional settings 
(Hammer, 1972 and Page, 1958). Other forms of feedback, such as knowledge of 
results, have also been shown to influence test performance (eg. Cummlngs, 
et>al., 1971 and Snyder, 1972) . These latter studies Indicate that knowledge 
of results Influences the goal setting behavior of students which, in turn, 
affects performance. 

Written comments may also function as stimuli for goal setting behavior. 
As Johnson (1973) suggests, students may try within the limits of their abilities 
to give the teacher what they think he wants. If this is true, It is reasonable 
to predict that coiments which focus directly on the "subjective'^ goals or 
stan'i> ' Is which students adopt will have a strong influence on test performance. 
This prediction is supported by Bsteban et .al . 's, 1972, finding that externally- 
induced goals promote achievement in a Verbal learning task. 

- OBJECTIVES - 

This study attempted to answer the following questions: 

1) Will written cornments on unit postests facilitate student achievement 
in a competency-based course? 



2) What are the differential effects, if any, of standard coiaments 
(e.g. "Good Show") and comments which are designed to encourage 
students to adopt a higher "subjective" standard of performance 
than that which is required by the course, (e,g. "I would like to 
see you set 13 as your mastery level,")? 

3) Can a set to work toward a higher "subjective" standard be estab- 
lished quickly and maintained throughout the course? 



METHOD - 



The 88 subjects in this study were enrolled in two sections of a 
graduate education course at a large Midwestern University during Summer Term, 
1973. Eighty-six percent of the students were ac<:ive teachers during the 
regular school year. The course itself consisted of eight distinct units. A 
complete list of behavioral objectives, written and verbal forms of instruction, 
a practice test, and alternate forms of a postest were carefully prepared for 
each unit. Those students who received scores of 12 or higher (80% accuracy) 
on a given unit postest received credit for that unit; those who scored below 
12 received remedial instruction and attempted the alternate form of the unit 
exam. In general, the only form of feedback which students received on the 
first administration of a given postest was the number of correct answers and 
a written comment, or no comment according to assignment to experimental groups. 
However, the instructor reviewed specific items with individual students who 
requested this form of feedback. 

Students in each section were randomly assigned to one of four experimental 
groups. There were 22 students In each group. 



Experimental Croup I : Higher Standard -Five treatments (H.S, - 5) 

Students In this group received written comments on the first five postests 
which encouraged them to adopt a higher "subjective" standard than that required 
for mastery of each unit In the course. A different set of comments was used 
for each postest and comments were varied according to the student's level of 
performance. Students who scored 13 or higher on the first postest* for example 
received the following comment, '^Your score suggests that you should set a hlghe 
standard such as 13 as your mastery level/V The comment, "I would like to see 



you set 13 as your mastery level" was written on the postest of students with 
scores of 12 or 12^^, while the conunent , "You might profit from studying for each 
test as If 13 were the nvistery leveV* was given to s tulents who fell below the 
established standard of 807, accuracy.* No cotnTOnts were written on postests 6-8* 

Experimental Group II ? Higher Standard - Three Treatments (H,S, - 3) 

Comments Cor students in this group were determined by the same policies 
as those used for Group I (!i.S. - 5) • However, conments were made on only the 
first three postests; no cornments were written on postests ^-8« 



Experimental Group III : Standard Comments, 

Students in this group received standard written comments on the first 
five unit postests. The same set of comments was used for each postest. The 
comments for the three achievement levels identified above were: **Good show, 
'^Close, but you made it . . and ^'Don^t get discouraged , . " No comments 
were written on postests 6-8, 



Experimental G roup 11^ 

Students in this group did not receive written comments on any of the 
eight unit postests. 



- ANALYSIS - 



Three measures of student achievement were determined: 

1) Sum of scores on Unit Postests 6^8 (Total possible - 40)^'''^ 

2) CoToprehenslve Examination : This exam consisted of six new Items for 
each of the first six units of the course. The exam was unannounced 

and was administered on the final day of instruction. (Total possible»36) 

3) Delay Score : Students could elect to take any postest at their 
convenience. The delay score was determined by adding the total number 
of Ixistructional days which elapsed between day one of the course and 

a student's first attempt to pass each of the final seven postests / 
Hence this score could theoretically range from a low of seven for a 
student who completed all seven postests on the first day of the course 
to a high of 175 (7 x 25) for a student who took all seven postests 
on the final (25th) day of the course. 



* The complete list of conunents for each experimental gro\i|> Is presented In 
the Appendix to this paper* 

The postest for Unit 7 consisted of 10 mtiltiple choito ttrms. All orht^r 
unit postests contained 15 items. 



A 2x4 (classes x treatments) multivariate analysis of covariance was 
computed as well as a univariate analysis of covariance for each of the three 
variables identified above. A student *s score on the first unit postest and the 
number of days which had elapsed before a student attempted the first postest 
served as the covartates. 

- RESULTS AND CONCLUSIONS - 

Differences in means for the two classes were not large and were not 
statistically significant for any of the three dependent variables: (p< .23) for 
the sum of scores on postests 6-fc8; (p<.82) for the comprehensive exam; and 
(p<«57) for delay scores. An omnibus test of Interaction effects (classes x 
treatments) was also not statistically significant: (p<.38) for the sum of 
scores on postests ; (p < .59) for the comprehensive exam; and (p<.86) for delay 
scores. Therefore the data for the two classes was combined. 

Table I depicts the adjusted means for the four treatment groups for 
each dependent variable. F - ratios and corresponding probability statements 
are also shown, Differences in mean scores are comparatively small and are not 
distributed in a consistent pattern across dependent variables . l^one of the 
differences is statistically significant. 



TABLE 1: ADJUSTED MEANS AND CORRESPONDING F-RATIOS FOR 
GROUPS ON THREE DEPENDENT VARUBLES 


THE FOUR TREATMENT 


Treatments 










Dependent 
Variable 


Exp. Grp. I 
(H.S.-5) 


Exp. Grp. II 
(H.S,-3) 


Exp.Grp.III 
(S.C.) 


Exp, Grp. IV 
(N.C.) 


Corresponding 
F-Ratto 


Sum of scores 
on Postests 6-8 


34.628 


35.949 


36.205 


35,992 


[,76 

(P < .16) 


Comprehensive 
Exam 


28. 6U 


27.045 


28,685 


28.342 


.88 
(p<.46) 


J Delay Scores 


123.996 


124.411 


125.023 1 

i 


121.526 


.53 
(p< .66) 



- CONCLUSIONS - 



The results support the following conclusions regarding the three questions 
which were posed in this study, 

1) Written comments on unit postests will not facilitate student 
achievement in a competency-based course at the graduate level, 

2) Different forms of written comments will not have differential 
effects on student achievement. 

3) It Is meaningless to question how quickly a "mental set'* can be 
established to work toward higher ^^subjective" standards . 

However, certain limitations should be considered prior to any attempt 

to generalize from the disappointing findings of this study. First, graduate 

students may be so highly motivated or so conditioned to receiving comments that 

any attempt to provide incentives in the form of comments may be ineffective. 

This suggestion is consistent with Harmner's (1972) failure to find significant 

differences between "standard comments** and "no comments'V f or college students* 

Second, the cotunents designed to encourage higher "subjective" standards may 

not have been understood by all students. Several students, for oxainple, 

asked the instructor to interpret the meaning of these statements. 

Finally, the failure to find differences in achievement does not suggest 
that comments serve no useful purpose. The results of a brief questionnaire 
which was administered to approximately 757c. of the students in both classes 
suggests that comments may influence student attitudes, 927^ of the students 
in this sample , for example , could correctly recall at least one of the comments 
they received. Further , ^6 these comments had a positive 

influence on their attitude towatd the course, while only 33y(. felt that the 
comments had a facilitating effect on their per f orrjance , 



- KDUCATIONAL IMPORTANCK OF THE STUDY - 

This study was designed to provide evidence relating to a crucial question 
confronting any Instructor of a competen-y-based course. When performance 
standards are deliberately set at a omparatlve ly low level in order to maximize 
student success, can higher levels of student achievement be maintained through 
other Instructional means? In view of their convenience and apparent effect, an 
instructor might attempt to use written comments to encourage sttident achievement. 
However, the results of this study suggest that whereas this approach may influenr 
student attitudes, standard co;ments or cr^mments aimed at establishing higher 
subjective goals are apt to have little or no impact on student achievement In a 
competency-based course at the graduate level. Thus other forms of comments 
or other means of facilitating higher levels of student achievement must be sought 



- REFERENCES - 



Blumfeld, G*J., Bostow, G., & Waugh, R. "Effect of Criterion Referenced Testing 
Upon the Use of Remedial Exam Opportunities* " Paper presented at A.E.R.A. 
national Conference; New York, 1971. 

Cinnmings, L.L., Schwab, Donald P., and Rosen, Marc "Performance and Knowledge 
of Results as Determinants of Goal Setting/* Journal of Applied Psychology > 
55: 526-530; December, 1971. 

Cureton, Louise W. , "The History of Grading Practices." N.C.M.E. Iteasuretrtent in 
Education, 2: No. 4; May, 1971. 

Ebel, Robert L. ''Criterion-Referenced Measurements: Limitations/' School Review ^ 
79: 282-88; February, 1971. 

Estaban, Arbues A., Polaino - Lorente, A, and Rodriguez, Sacristan, J. "An 

Experlement on Stimulation, Performance, and Level of Aspiration." Revlsta 
de Pslcologla General y Apllcada 27 (114-115); 87-117; January, 1972. 

Hammer, B. ''Grade Expectation, Differential Teacher Comments, and Student Performance/' 
Journal of Educational Psychology 63; 454-58; October, 1972. 

Johnson, Donald Mv> "Teacher's Corwnents and Creative Achievement/* Unpublished 
manuscript, 1973. 

Mlllman, Jason, "Reporting Student Progress: A Case For Criterion-Referenced 
Marking System/'' Phi Delta Kappan 52: 226-30; December > 1970. 

Mlllman, Jason, "Passing Scores and Test Lengths for Domin-Referenced Measures." 
Reviev of Education Research 43; 205-16; Spring, 1973. 

Page, E»B*, "Teacher Comnents and Student Performance: A Seventy-four Classroom 
Experiment In SchoolMotlvatlon." Journal of Educational Psychology 49: 173-81; 
1958. 

Popham, W.H. and Husek, T.R. , "Implications of Criterion Referenced Measurement." 
Journal of Educational Measurement , 6: 1-9; 1969 



APPENDIX: COMMENtS PROVIDED IN EACH EXPERIMENTAL CONDITION 

I. For Students with Scores of 13 or Higher; 

A. EXPERIMENTAL GROUPS I and II: "Higher Subjective Standards" 

Postest I - ''Your score suggests you should set a higher standard 
such as 13 as your mastery level. 

Postest 2 - "Are you using 13 as your mastery leve 1? 

Postest 3 - "Remember * . . the magic mastery level is 13 

Postest 4 - (U.S. - 5 only) - ''A mastery level of 13 seems to 
be working for you." 

Postest 5 - (H.S. - V only) "Is 13 a high enough mastery level for you?" 

B. EXPERIMENTAL GROUP III: "Standard Coraments" 
Postest 1-5 "Good show * . . " 

II. For Students with Scores of 12 or 12^: ' 

A. EXPERIMENTAL GROUPS I and II: "Higher Subjective Standards" 

Postest 1 - 'I would like to see you set a score of 13 as your 
mastery level-" 

Postest 2 - "Can you hit a mastery level of 13 on the next test?" 

Postest 3 - "Remember . . » The magic mastery level is 13." 

Postest 4 - (H.S. ~ 5 only) "Hope you"* re still using 13 as your mastery 
level , 

Postest 5 - (H.S.- 5 only) "Are you thinking of a mastery level of 13?" 

B. EXPERIMENTAL GROUP III: "Standard Comments" 
Postest 1-5. "Close, but you made it ..." 

III. For Students with Scores of less than 12; 

A. EXPERIMENTAL GROUPS I and II: "Higher Subjective StandardB" 

Postest 1 - "You might profit from studying for tach teal an 1 f 1) 
wore I ho mafltcry Icvol," 

PoHtcst 2 - "An^ you , studying lor each lost u.t If ll wcro tJie nmntory Icvc 

Postest 3 - "Remember . , . the magic raastury level is 13." 

Postest 4 - (H,S.-5 only) - "Hope you're still aiming for a mastery level 
of 13 when you study*" 
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, Postest 5 - (H.S.-5 only) "Are you working toward a mastery level 
of 13 when you study?" 



B. EXPERIMENTAL GROUP III: "Standard Coranents" 



Postest 1-5. "Don't get discouraged 
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